Extracting multiple classified adaptive decimation filters

ABSTRACT

A plurality of initial decimation filters are concatenated into an initial combined decimation filter. The initial decimation filters are assigned to a plurality of classes which associate the initial decimation filters with regions of an image to be predicted. An input signal is generated from the initial combined decimation filter and is weighted with a plurality of prediction filters. A correlation matrix and an observation vector are generated for the initial combined decimation filter from the input signal. An optimized combined decimation filter is extracted from the correlation matrix and observation vector. The optimized filter comprises a plurality of optimized decimation filters. The optimized combined decimation filter is de-concatenated into the plurality of optimized decimation filters.

RELATED APPLICATIONS

The present application is related to the following commonly-owned,concurrently-filed applications: application Ser. No. ______ (AttorneyDocket No. 080398.P735), filed ______ entitled “Adaptive DecimationFilter.”

FIELD OF THE INVENTION

This invention relates generally to data processing, and moreparticularly to video encoding.

COPYRIGHT NOTICE/PERMISSION

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever. The following notice applies: Copyright© 2008, SonyElectronics Inc., All Rights Reserved.

BACKGROUND OF THE INVENTION

An important and fundamental problem in various video processingapplications, such as compression and zooming, is decimation andprediction. Decimation involves generating a smaller version of a sourcedata. A common example is video compression, which uses a decimationfilter to decimate an original frame in order to create a decimatedframe. To decompress the video frame, a prediction filter generates atarget frame from the decimated frame. The quality of the target framemay be expressed by measuring the difference between the original frameand the target frame, which is the prediction error. One existingprocess used to improve the quality of the target frame is to use anadaptive prediction filter. An adaptive prediction filter is modifiedwith respect to a corresponding decimated frame to improve the qualityof the target frame, while the decimation filter remains fixed. Aclassified prediction filter is a adaptive prediction filter thatapplies different filters/classes to different pixels/regions in animage.

FIG. 2A illustrates a fixed decimation filter with a decimation factorof ¼ that reduces the original frame to a decimated frame one quarterthe size of the original. Original pixels 205 (unfilled) are eachassigned a decimation filter coefficient of ¼. In other words, togenerate the decimated pixel 210, original pixels 205 will be averaged.FIG. 2B illustrates a fixed decimation filter with a decimation factorof 1/9. Original pixels 250 (unfilled, plus an original pixel in thesame location as target pixel 255) each contribute 1/9 to the value ofdecimated pixel 255. After applying the decimation filter illustrated inFIG. 2A or 2B, the source data has been reduced in size consistent withthe decimation factor.

SUMMARY OF THE INVENTION

A plurality of initial decimation filters are concatenated into aninitial combined decimation filter. The initial decimation filters areassigned to a plurality of classes which associate the initialdecimation filters with regions of an image to be predicted. An inputsignal is generated from the initial combined decimation filter and isweighted with a plurality of prediction filters. A correlation matrixand an observation vector are generated for the initial combineddecimation filter from the input signal. An optimized combineddecimation filter is extracted from the correlation matrix andobservation vector. The optimized filter comprises a plurality ofoptimized decimation filters. The optimized combined decimation filteris de-concatenated into the plurality of optimized decimation filters.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a overview of the operation of anembodiment of an adaptive decimation filter according to the invention;

FIGS. 2A and 2B are diagrams illustrating fixed decimation filters asknown in the art;

FIG. 3 is a diagram illustrating an adaptive decimation filter accordingto an embodiment of the invention;

FIG. 4 is a data flow diagram illustrating the operation of an adaptivedecimation filter according to an embodiment of the invention;

FIG. 5 is a flow diagram of a filter optimization method to be performedby a computer according to an embodiment of the invention;

FIG. 6 is a diagram illustrating the operation of an adaptive decimationfilter according to an embodiment of the invention;

FIGS. 7A and 7B are flow diagrams of filter optimization methods to beperformed by a computer according to embodiments of the invention;

FIG. 8 is a diagram illustrating the operation of an adaptive decimationfilter according to an embodiment of the invention;

FIG. 9 is a diagram illustrating the operation of an adaptive decimationfilter according to an embodiment of the invention:

FIG. 10 is a diagram illustrating the operation of an adaptivedecimation filter according to an embodiment of the invention;

FIGS. 11A and 11B are diagrams of a computer environment suitable forpracticing the invention;

FIG. 12 is a diagram of a classified adaptive decimation filteraccording to an embodiment of the invention;

FIG. 13 is a flow diagram of a multiple decimation filter extractionmethod to be performed by a computer according to an embodiment of theinvention;

FIG. 14 is a flow diagram of another multiple decimation filterextraction method to be performed by a computer according to anembodiment of the invention; and

FIG. 15 is a flow diagram of an initial estimation of decimation filtersmethod to be performed by a computer according to an embodiment of theinvention.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description of embodiments of the invention,reference is made to the accompanying drawings in which like referencesindicate similar elements, and in which is shown by way of illustrationspecific embodiments in which the invention may be practiced. Theseembodiments are described in sufficient detail to enable those skilledin the art to practice the invention, and it is to be understood thatother embodiments may be utilized and that logical, mechanical,electrical, functional and other changes may be made without departingfrom the scope of the present invention. The following detaileddescription is, therefore, not to be taken in a limiting sense, and thescope of the present invention is defined only by the appended claims.

Beginning with an overview of the operation of the invention, FIG. 1illustrates one embodiment of adaptive decimation filtering. Originaldata 100 is encoded at encoder 105 using adaptive prediction anddecimation filters. The original data 100 may be video data from aBluRay player, which a user wishes to view on a standard definitiontelevision such as device 120. To accomplish this, encoder 105 may zoomdata 100 to produce zoomed data 110, which device 120 is able to decodeand display at the appropriate aspect ratio. The encoded data fromencoder 105 may take the form of compressed data 115, which encoder 105may cause to be sent over network 125 to device 130 to decode anddisplay. For example, data 100 may be live video of a sporting event.

The predictive strength of a prediction filter or a decimation filtermay understood in terms of how similar an original frame is to areconstruction of that original frame using the prediction filter.Optimizing either a prediction filter or a decimation filter is theprocess of adjusting the coefficients that make tip the filter in a waythat decreases the difference between the source frame and the targetframe. In other words, reducing the prediction error associated with thetarget frame.

Optimizations to a decimation filter are present, indirectly, in thetarget frame by way of the decimated frame that the prediction filteruses to generate the target frame. As an example, suppose a predictionfilter L is used to generate two target frames, P and P′. To generatethe target frames, L is applied to two decimated frames, D and D′, whichwere generated by two decimation filters, H and H′. Further suppose thatthe first target frame, P, has an average prediction error of 0.05 whencompared to original image I, while the second target frame P′ has anaverage prediction error of 0.10 when compared to I. Since the sameprediction filter L, is used to produce both target frames, thedifference in accuracy is attributable to the decimated frames and thusthe different decimation filters.

FIG. 4 is a data flow diagram illustrating the operation of an adaptivedecimation filter according to an embodiment of the invention. Encodercontext 400 includes original frames 410 and decimated frames 425, whichare decimated 415 by an encoder such as encoder 105 and predicted 420 toimprove a prediction by measuring the error of the prediction. Encodeddata 430 is sent to decoder 405. For example, over network 125. Decoder105 uses decimated frames 435 and filter coefficients also included inthe encoded data 430 to predict 440 predicted frames 445. Predictedframes 445 may be displayed to a user.

Next, the particular methods of the invention are described in terms ofcomputer software with reference to a series of flow diagrams. FIGS. 5and 7 are flow diagrams illustrating methods according to embodiments ofan adaptive decimation filter. Encoder 105 may perform the methodsillustrated in FIGS. 5 and 7. The methods constitute computer programsmade up of machine-executable instructions illustrated as blocks (acts)505 until 525 in FIG. 5 and blocks 705 to 740 in FIG. 7. Describing themethods by reference to a flow diagram enables one skilled in the art todevelop such programs including such instructions to carry out themethods on suitably configured machines (the processor of the machineexecuting the instructions from machine-readable storage media,including memory such as RAM) [computerized clients (the processor ofthe clients executing the instructions from machine-readable media).Similarly, the methods to be performed by the server constitute computerprograms also made up of machine-executable instructions. Describing themethods by reference to flow diagrams enables one skilled in the art todevelop programs including instructions to carry out the methods on asuitable computerized server (the processor of the server executing theinstructions from machine-readable storage media).] Themachine-executable instructions may be written in a computer programminglanguage or may be embodied in firmware logic. If written in aprogramming language conforming to a recognized standard, suchinstructions can be executed on a variety of hardware platforms and forinterface to a variety of operating systems. In addition, the presentinvention is not described with reference to any particular programminglanguage. It will be appreciated that a variety of programming languagesmay be used to implement the teachings of the invention as describedherein. Furthermore, it is common in the art to speak of software, inone form or another (e.g., program, procedure, process, application,module, logic . . . ), as taking an action or causing a result. Suchexpressions are merely a shorthand way of saying that execution of thesoftware by a computer causes the processor of the computer to performan action or produce a result. It will be appreciated that more or fewerprocesses may be incorporated into the methods illustrated in FIGS. 5and 7 without departing from the scope of the invention and that noparticular order is implied by the arrangement of blocks shown anddescribed herein.

FIG. 3 is a diagram illustrating adaptive decimation filtering accordingto an embodiment of the invention. Pixel tap 305 includes pixel regions325 and 330 and corresponds, through the decimation filter, to decimatedpixel 310. To improve the predictive strength of the decimation filter,encoder 105 may optimize the decimation filter by changing the number ofpixels included in a pixel tap or the coefficients corresponding topixels in a pixel tap used to generate a decimated pixel. For example,in a fixed decimation filter with a decimation factor of ¼, thedecimation filter may use a pixel tap such as pixel region 325. Encoder105 may expand this initial tap to create pixel tap 305, whichincorporates additional pixels into the tap. Pixels in region 325 mayhave coefficients of 0.125, while pixels in region 330 may havecoefficients of 0.05, and pixels in corners of the tap may havecoefficients of 0.025. By increasing the size of pixel tap 305, moreoriginal data is available to generate decimated pixel 310, which inturn may improve the quality of a target frame. The encoder may alsovary the filter coefficients according to which original pixels producea better prediction.

For example, encoder 105 may also modify a pixel tap to exploitunderlying features of the original frame. Pixel tap 315, for example,is elongated in the horizontal direction, and corresponds to decimatedpixel 320. The region of the original frame from which pixel tap 315 isdrawn may contain a horizontal feature (e.g., a brick in a wall), inwhich case a horizontally elongated pixel tap such as pixel tap 315 mayproduce a decimated pixel 320 which provides greater predictive strengthto a prediction filter.

FIG. 5 illustrates a filter optimization method according to anembodiment of the invention. Encoder 105 may perform filter optimizationmethod 500 to optimize an adaptive decimation filter and an adaptiveprediction filter. One obstacle to optimizing both the decimation filterand the prediction filter is that a simultaneous optimization would be anon-linear process. The prediction filter depends on the decimatedframe, which is obtained with the decimation filter. An optimizeddecimation filter provides high prediction strength, where predictionstrength depends directly on the prediction filter. The non-linearity ofthe relationship may be expressed as a function of the prediction anddecimation filters:

I ^(D) =H·I   (1)

where I^(D) is the decimated frame. H is the decimation filter, and I isthe original frame. The prediction filter may be defined as:

P=L(H)·I   (2)

where P is a target frame and prediction filter L is a function of H.Thus, target frame P may be expressed as:

P=L(H)·H·I   (3)

and prediction error E may be expressed as:

E=(I−P)²=(I−L(H)·H·I)²   (4)

which is a nonlinear function in L and H. To obtain optimal predictionand decimation filters, prediction error E is minimized. Since the errorfunction is nonlinear, no direct method to extract optimal solutions forboth the prediction and decimation filters is available.

In the embodiment illustrated by FIG. 5, a recursive process is used tooptimizing the prediction and decimation filters. This process uses alinear error function. At block 505, the method 500 selects an initialdecimation filter. For example, initial decimation filter may be a fixeddecimation filter. At block 515, the method 500 extracts an optimizedprediction filter while holding the decimation filter fixed. By keepingthe decimation filter fixed, the error function becomes quadratic. Thus,minimizing the prediction error yields linear equations for a solutionof the filter, which allows the method to extract an optimizedprediction filter for the current decimation filters Extracting anoptimized prediction filter is described in greater detail below inconjunction with FIG. 7.

At block 520, the method 500 extracts an optimized decimation filterwhile holding the prediction filter extracted in block 515 fixed. Byholding the prediction filter fixed, the error function becomesquadratic, allowing the method to extract an optimized decimation filterfor the current prediction filter. If at decision 525 the methoddetermines the optimization is complete, the method returns control to,for example, encoder 105. Otherwise, the method returns to block 515 tofurther optimize the filters.

By using the recursive process described above, the method 500incrementally improves both the decimation filter and the predictionfilter. Since prediction strength depends upon the adaptability of bothfilters, changes to one filter may make new improvements possible in theother filter.

FIG. 6 illustrates the operation of the filter optimization methodillustrated in FIG. 5. Passes 600, 630, and 655 represent sequentialoptimizations of prediction and decimation filters. The method 500decimates original frame 605 with an initial decimation filter 610 toproduce decimated frame 620. The method 500 extracts an optimizedprediction filter 615 from decimated frame 620, as described in greaterdetail below in conjunction with FIG. 7. During pass 630, the method 500extracts an optimized decimation filter 640 from original frame 605while holding prediction filter 615 fixed. The method 500 generates adecimated frame 650 with decimation filter 640 and extracts an optimizedprediction filter 645 while holding decimation filter 640 fixed. Duringpass 655, the method 500 extracts an optimized decimation filter 665while holding prediction filter 645 fixed. The method 500 generatesdecimated frame 675 using decimation filter 665. The method 500 extractsan optimized prediction filter 670 using decimated frame 675. The method500 determines that the results of optimization have saturated andrelinquishes control, for example, to encoder 105.

FIG. 7 illustrates a method to be performed according to an embodimentof the invention. At block 705, the method 700 selects an initialdecimation filter, such as a fixed decimation filter with the desireddecimation factor. At block 710, the method 700 generates a decimatedframe by applying the decimation filter to an original frame (e.g., aframe of video). At block 715, the method 700 generates classes for aprediction filter using the decimated frame using one of many processesknown in the art. At block 720, the method 700 extracts a classifiedprediction filter while holding the decimation filter fixed. The methodgenerates an optimized prediction filter by keeping the decimationfilter as a fixed variable.

The notation used to describe the filter extraction process isdescribed, followed by the process itself. The original frame signallevel is denoted for an individual pixel {right arrow over (x)} asI({right arrow over (x)}). The decimated frame signal level is denotedfor the decimated pixel {right arrow over (y)} as I^(D)({right arrowover (y)}), while the set of decimated pixels defined on the predictionfilter tap for the prediction of target pixel x, where the j index runsover the filter tap of the prediction filter is {right arrow over(y)}_(j) ^({right arrow over (x)}).

The prediction of the original frame for the pixel x is P({right arrowover (x)}) and is defined in the following paragraph. The predictionfilter corresponding to the prediction filter class u is denoted asL^(u).

The prediction filter is derived from the minimization of the predictionerror, holding the decimation filter constant. The prediction for thepixel x may be defined as:

$\begin{matrix}{{P\left( \overset{\rightarrow}{x} \right)} = {\sum\limits_{j}{L_{j}^{u}{I^{D}\left( {\overset{\rightarrow}{y}}_{j}^{\overset{\rightarrow}{x}} \right)}}}} & (5)\end{matrix}$

while the prediction filter class for the pixel x may be defined as:

u=S({right arrow over (x)})   (6)

while the prediction error may be defined as:

$\begin{matrix}{e = {\overset{\;}{\sum\limits_{\overset{\rightarrow}{x}}}\left( {{I\left( \overset{\rightarrow}{x} \right)} - {P\left( \overset{\rightarrow}{x} \right)}} \right)^{2}}} & (7)\end{matrix}$

A least squares training solution involves computing a correlationmatrix and observation vector over all pixels {right arrow over (x)}such that S({right arrow over (x)})=u. The least squares solution yieldsa stable solution when the set of pixels is large enough for each filterclass. The method 700 extracts the prediction filter L^(v) for someclass v from prediction error minimization with fixed decimation using aleast squares solution:

$\begin{matrix}{{\frac{\delta \; e}{\delta \; L_{k}^{v}}}_{H} = 0} & (8) \\{\sum\limits_{\overset{\rightarrow}{x}}^{\;}{\left( {{I\left( \overset{\rightarrow}{x} \right)} - {P\left( \overset{\rightarrow}{x} \right)}} \right){\sum\limits_{j}{\frac{\delta \; L_{j}^{u}}{\delta \; L_{k}^{v}}{I^{D}\left( {\overset{->}{y}}_{j}^{\overset{->}{x}} \right)}}}}} & (9) \\{{\sum\limits_{\{{{\overset{->}{x}|v} = {S(\overset{->}{x})}}\}}^{\;}{\left( {{I\left( \overset{\rightarrow}{x} \right)} - {P\left( \overset{\rightarrow}{x} \right)}} \right){I^{D}\left( {\overset{->}{y}}_{k}^{\overset{->}{x}} \right)}}} = 0} & (10) \\{{\sum\limits_{\{{{\overset{->}{x}|v} = {S(\overset{->}{x})}}\}}^{\;}{{I\left( \overset{\rightarrow}{x} \right)}{I^{D}\left( {\overset{->}{y}}_{k}^{\overset{->}{x}} \right)}}} = {\sum\limits_{\{{{\overset{->}{x}|v} = {S(\overset{->}{x})}}\}}^{\;}{\sum\limits_{j}{L_{j}^{v}{I^{D}\left( {\overset{->}{y}}_{j}^{\overset{->}{x}} \right)}{I^{D}\left( {\overset{->}{y}}_{k}^{\overset{->}{x}} \right)}}}}} & (11) \\{b_{k}^{v} = {\sum\limits_{j}{L_{j}^{v}A_{j,k}^{v}}}} & (12)\end{matrix}$

The optimized prediction filter for class v may be given by:

L ^(v)=(A ^(v))⁻¹ b ^(v)   (13)

where

$\begin{matrix}{A_{j,k}^{v} = {\sum\limits_{\{{{\overset{->}{x}|v} = {S(\overset{->}{x})}}\}}^{\;}{{I^{D}\left( {\overset{->}{y}}_{j}^{\overset{->}{x}} \right)}{I^{D}\left( {\overset{->}{y}}_{k}^{\overset{->}{x}} \right)}}}} & (14)\end{matrix}$

is the correlation matrix for class v and

$\begin{matrix}{b_{k}^{v} = {\sum\limits_{\{{{\overset{->}{x}|v} = {S(\overset{->}{x})}}\}}^{\;}{{I\left( \overset{\rightarrow}{x} \right)}{I^{D}\left( {\overset{->}{y}}_{k}^{\overset{->}{x}} \right)}}}} & (15)\end{matrix}$

is the observation vector for class v, resulting in v independent matrixequations, one for each class in the filter. The indexes {j, k } runover the prediction filter taps

j=1,2, . . . T₁,k=1,2 . . . T₁   (16)

with T₁ denoting the size of the prediction filter tap.

At block 725, the method 700 generates a predicted frame using theprediction filter and the decimated frame.

At block 730, the method 700 generates classes for the decimation filterfrom the target frame. The decimation classes may be defined on theoriginal input data, or on the decimated data itself The followingprocess applies to deriving an adaptive decimation filter using a singleclass (mono-class). Deriving a multi-class adaptive decimation filter isdescribed in greater detail below in conjunction with FIGS. 13-15.

At block 735, the method 700 extracts a classified decimation filterwhile holding the prediction filter fixed. The method 700 derives thedecimation filter from the minimization of the prediction error, keepingthe prediction filter fixed. The prediction for the pixel x may bedefined as:

$\begin{matrix}{{P\left( \overset{\rightarrow}{x} \right)} = {\sum\limits_{j}{L_{j}^{u}{I^{D}\left( {\overset{->}{y}}_{j}^{\overset{->}{x}} \right)}}}} & (17)\end{matrix}$

the prediction filter class map for the pixel; may be defined as,

u=S({right arrow over (x)})   (18)

the decimated image may be defined as:

$\begin{matrix}{{I^{D}\left( y_{j}^{\overset{->}{x}} \right)} = {\sum\limits_{n}{H_{u}{I\left( {\overset{->}{z}}_{n}^{{\overset{->}{y}}_{j}^{\overset{\cdot}{x}}} \right)}}}} & (19)\end{matrix}$

the prediction for the pixel ; expressed in terms of the decimationfilter may be defined as:

$\begin{matrix}{{P\left( \overset{->}{x} \right)} = {\sum\limits_{j}{L_{j}^{u}{\sum\limits_{n}{H_{n}{I\left( {\overset{->}{z}}_{n}^{{\overset{->}{y}}_{j}^{\overset{\cdot}{x}}} \right)}}}}}} & (20)\end{matrix}$

and the prediction error may be defined as:

$\begin{matrix}{e = {\overset{\;}{\sum\limits_{\overset{\rightarrow}{x}}}\left( {{I\left( \overset{\rightarrow}{x} \right)} - {P\left( \overset{\rightarrow}{x} \right)}} \right)^{2}}} & (21)\end{matrix}$

Accordingly, the method 700 extracts the decimation filter H fromprediction error minimization with a fixed prediction filter, asfollows:

$\begin{matrix}{\mspace{76mu} {\left. \frac{\delta \; e}{\delta \; H_{m}} \right|_{L} = 0}} & (22) \\{\mspace{70mu} {{\sum\limits_{\overset{\rightarrow}{x}}{\left( {{I\left( \overset{\rightarrow}{x} \right)} - {P\left( \overset{\rightarrow}{x} \right)}} \right){\sum\limits_{j}{L_{j}^{u}{\sum\limits_{n}{\frac{\delta \; H}{\delta \; H_{m}}{I\left( {\overset{\rightarrow}{z}}_{n}^{{\overset{\rightarrow}{y}}_{j}^{\overset{\rightarrow}{x}}} \right)}}}}}}} = 0}} & (23) \\{\mspace{70mu} {{\sum\limits_{\overset{\rightarrow}{x}}{\left( {{I\left( \overset{\rightarrow}{x} \right)} - {P\left( \overset{\rightarrow}{x} \right)}} \right){\sum\limits_{j}{L_{j}^{u}{I\left( {\overset{\rightarrow}{z}}_{m}^{{\overset{\rightarrow}{y}}_{j}^{\overset{\rightarrow}{x}}} \right)}}}}} = 0}} & (24) \\{{\sum\limits_{\overset{\rightarrow}{x}}{\sum\limits_{j}{{I\left( \overset{\rightarrow}{x} \right)}L_{j}^{u}{I\left( {\overset{\rightarrow}{z}}_{m}^{{\overset{\rightarrow}{y}}_{j}^{\overset{\rightarrow}{x}}} \right)}}}} = {\sum\limits_{\overset{\rightarrow}{x}}{\sum\limits_{j}{\sum\limits_{k}{L_{k}^{u}{\sum\limits_{n}{H_{n}{I\left( {\overset{\rightarrow}{z}}_{n}^{{\overset{\rightarrow}{y}}_{k}^{\overset{\rightarrow}{x}}} \right)}L_{j}^{u}{I\left( {\overset{\rightarrow}{z}}_{m}^{{\overset{\rightarrow}{y}}_{j}^{\overset{\rightarrow}{x}}} \right)}}}}}}}} & (25) \\{\mspace{76mu} {{\sum\limits_{\overset{\rightarrow}{x}}{{I\left( \overset{\rightarrow}{x} \right)}{F_{m}\left( \overset{\rightarrow}{x} \right)}}} = {\sum\limits_{n}{H_{n}{\sum\limits_{\overset{\rightarrow}{x}}{{F_{n}\left( \overset{\rightarrow}{x} \right)}{F_{m}\left( \overset{\rightarrow}{x} \right)}}}}}}} & (26) \\{\mspace{76mu} {where}} & \; \\{\mspace{70mu} {{F_{m}\left( \overset{\rightarrow}{x} \right)} = {\sum\limits_{j}{L_{j}^{u}{I\left( {\overset{\rightarrow}{z}}_{m}^{{\overset{\rightarrow}{y}}_{j}^{\overset{\rightarrow}{x}}} \right)}}}}} & (27)\end{matrix}$

The decimation filter H is given by:

H=(A)⁻¹ b   (28)

where

$\begin{matrix}{A_{n,m} = {\sum\limits_{\overset{\rightarrow}{x}}{{F_{n}\left( \overset{\rightarrow}{x} \right)}{F_{m}\left( \overset{\rightarrow}{x} \right)}}}} & (29)\end{matrix}$

defines the correlation matrix and

$\begin{matrix}{b_{m} = {\sum\limits_{\overset{\rightarrow}{x}}{{I\left( \overset{\rightarrow}{x} \right)}{F_{m}\left( \overset{\rightarrow}{x} \right)}}}} & (30)\end{matrix}$

defines the observation vector. The indexes {m,n} run over thedecimation filter taps:

m=1,2, . . . T ₂ , n=1,2 . . . T ₂   (3)

with T₂ denoting the size of the decimation filter tap.

The method 700 generates the decimation filter H using the least squaressolutions above, given the fixed classified prediction filter. Thefunction F_(n)({right arrow over (x)}) incorporates the effect of theclassified prediction filter which the method 700 generates at block720. The function F_(n)({right arrow over (x)}) provides a weightedaverage of the original data and the classified prediction filter L fromblock 720. The weighted average is over all of the tap positions {j} ofthe classified prediction filter.

$\begin{matrix}{{F_{n}\left( \overset{\rightarrow}{x} \right)} = {\sum\limits_{j}{L_{j}^{S{(\overset{\rightarrow}{x})}}\left( {\overset{\rightarrow}{z}}^{{\overset{\rightarrow}{y}}_{j}^{\overset{\rightarrow}{x}}} \right)}}} & (32)\end{matrix}$

The function F_(n)({right arrow over (x)}) may be referred to as theL-weighted input signal. At decision 740, the method 700 determineswhether to further optimize the filters or to terminate. In oneembodiment, the method 700 may repeat the process a predetermined numberof times. Empirically, the result saturates after 4 or 5 passes using adecimation factor of 1/9, so the method 700 may terminate after 4 or 5passes.

FIG. 8 is a diagram illustrating the optimization of a two-dimensional(2D) prediction filter and a declination filter. The prediction filteris 2D in that it generates a predicted frame using only decimated pixelsfrom the same time instant as the target (predicted) frame. Decimationfilter 810 generates decimated frame 805 from original frame 800. 2Dprediction filter 815 uses decimated pixel tap 810 to reconstruct targetpixel 820.

FIG. 9 is a diagram illustrating the optimization of a three-dimensional(3D) prediction filter and decimation filter forprogressive/non-interlaced video frames. Decimation filters 905 and 935generate decimated frames 910 and 945 from original frames 900 and 925.Prediction filter 920 is 3D in that it uses decimated taps in thedecimated frame corresponding to the target frame and in at least onedecimated frame offset in time from the target frame. For example,prediction filter 920 uses decimated pixel tap 915 from offset decimatedframe 910 and decimated pixel tap 950 in decimated frame 945 toreconstruct target pixel 930.

FIG. 10 is a diagram illustrating the optimization of three-dimensional(3D) prediction filters and decimation filters for interlaced video datawhere the frame is separated into two fields, offset vertically.Generally, the two fields from a given frame are separated in time by1/60 of a second and are shifted vertically by one row of pixels. Eachframe of video data is represented by two fields of interlaced videodata. In FIG. 10, fields 1005 and 1015 form the target frame. In thisembodiment, each field has a corresponding decimation filter. Fields1005 and 1015 in the target frame have a corresponding 3D predictionfilter. Prediction filter 1065 generates target pixel 1045 usingdecimated pixel taps 1035 and 1040 in decimated fields 1020 and 1025.Decimated field 1025 corresponds chronologically to the target field1005. Decimated field 1020 corresponds chronologically to the secondfield of the previous frame of video data. Prediction filter 1070generates target pixel 1060 using decimated pixel taps 1055 and 1050.Decimated pixel tap 1055 corresponds chronologically to the target field1015, while decimated pixel frame 1050 corresponds chronologically tothe previous field 1005 in the target frame.

FIG. 7B is a flow diagram illustrating a method of optimizing classifiedadaptive decimation filters according to an embodiment of the invention.FIGS. 7A and 7B in that FIG. 713 incorporates block 730. At block 730,the method 750 generates classes for the decimation filter. Also, atblock 735, the method extracts a classified filter using the classes.Prediction using classified decimation filters may be defined as:

$\begin{matrix}{{P\left( \overset{\rightarrow}{x} \right)} = {\sum\limits_{j}{L_{j}^{u}{I^{D}\left( {\overset{\rightarrow}{y}}_{j}^{\overset{\rightarrow}{x}} \right)}}}} & (33)\end{matrix}$

where the prediction map is given by:

u=S({right arrow over (x)})   (34)

The decimated image is given by:

$\begin{matrix}{{I^{D}\left( y_{j}^{\overset{\rightarrow}{x}} \right)} = {\sum\limits_{n}{H_{n}^{v}{I\left( {\overset{\rightarrow}{z}}_{n}^{{\overset{\rightarrow}{y}}_{j}^{\overset{\rightarrow}{x}}} \right)}}}} & (35)\end{matrix}$

The decimation class map is:

v=D({right arrow over (y)} _(j) ^({right arrow over (x)}))   (36)

The prediction may be expressed in terms of the decimation filter as:

$\begin{matrix}{{P\left( \overset{\rightarrow}{x} \right)} = {\sum\limits_{j}{L_{j}^{u}{\sum\limits_{n}{H_{n}^{v}{I\left( {\overset{\rightarrow}{z}}_{n}^{{\overset{\rightarrow}{y}}_{j}^{\overset{\rightarrow}{x}}} \right)}}}}}} & (37)\end{matrix}$

The prediction error is the difference between the image to be predictedand the prediction:

$\begin{matrix}{e = {\sum\limits_{\overset{\rightarrow}{x}}\left( {{I\left( \overset{\rightarrow}{x} \right)} - {P\left( \overset{\rightarrow}{x} \right)}} \right)^{2}}} & (38)\end{matrix}$

The prediction error is a function of the prediction and decimationfilters, and the prediction and decimation class map. This dependency isexpressed as e(L,H,C,D). The class maps are usually fixed and determinedby some other method, so the dependency of the prediction error on thefilters only is expressed as: e(L,H). In other embodiments otherclassification methods may be used.

The multiple classified decimation filters {H^(v),v=0,1,2 . . . M−1} arederived from the minimization of the prediction error. This minimizationis obtained by setting the variation of the prediction error to zerowhile keeping the prediction error fixed:

$\begin{matrix}{\left. \frac{\delta \; {e\left( {L,H} \right)}}{\delta \; H_{m}^{t}} \right|_{L} = 0} & (39)\end{matrix}$

Therefore, the mono-class decimation described above primarily inconjunction with FIG. 7 may be extended to multi-class decimation usingthe following process:

$\begin{matrix}{{\sum\limits_{\overset{\rightarrow}{x}}{\left( {{I\left( \overset{\rightarrow}{x} \right)} - {P\left( \overset{\rightarrow}{x} \right)}} \right){\sum\limits_{j}{L_{j}^{u}{\sum\limits_{n}{\frac{\delta \; H_{n}^{v}}{\delta \; H_{m}^{t}}{I\left( {\overset{\rightarrow}{z}}_{n}^{{\overset{\rightarrow}{y}}_{j}^{\overset{\rightarrow}{x}}} \right)}}}}}}} = 0} & (40) \\{{\sum\limits_{\overset{\rightarrow}{x}}{\left( {{I\left( \overset{\rightarrow}{x} \right)} - {P\left( \overset{\rightarrow}{x} \right)}} \right){\sum\limits_{\{{{j|t} = {D{(y_{j}^{\overset{\rightarrow}{x}})}}}\}}{L_{j}^{u}{I\left( {\overset{\rightarrow}{z}}_{m}^{{\overset{\rightarrow}{y}}_{j}^{\overset{\rightarrow}{x}}} \right)}}}}} = 0} & (41)\end{matrix}$

Applying the prediction defined in equation 37, the following isobtained for equation 40:

$\begin{matrix}{{\sum\limits_{\overset{\rightarrow}{x}}{\sum\limits_{\{{{j|t} = {D{(y_{j}^{\overset{\rightarrow}{x}})}}}\}}{{I\left( \overset{\rightarrow}{x} \right)}L_{j}^{u}{I\left( {\overset{\rightarrow}{z}}_{m}^{{\overset{\rightarrow}{y}}_{j}^{\overset{\rightarrow}{x}}} \right)}}}} = {\sum\limits_{\overset{\rightarrow}{x}}{\sum\limits_{k}{L_{k}^{u}{\sum\limits_{n}{H_{n}^{v}{I\left( {\overset{\rightarrow}{z}}_{n}^{{\overset{\rightarrow}{y}}_{k}^{\overset{\rightarrow}{x}}} \right)}{\sum\limits_{\{{{j|t} = {D{(y_{j}^{\overset{\rightarrow}{x}})}}}\}}{L_{j}^{u}{I\left( {\overset{\rightarrow}{z}}_{m}^{{\overset{\rightarrow}{y}}_{j}^{\overset{\rightarrow}{x}}} \right)}}}}}}}}} & (42)\end{matrix}$

Which may also be expressed as:

$\begin{matrix}{{\sum\limits_{\overset{\rightarrow}{x}}{{I\left( \overset{\rightarrow}{x} \right)}{F_{m}^{t}\left( \overset{\rightarrow}{x} \right)}}} = {{\sum\limits_{n}{H_{n}^{t}{\sum\limits_{\overset{\rightarrow}{x}}{{F_{n}^{t}\left( \overset{\rightarrow}{x} \right)}{F_{m}^{t}\left( \overset{\rightarrow}{x} \right)}}}}} + {G_{m}^{t}\left( \left\{ H^{v \neq t} \right\} \right)}}} & (43)\end{matrix}$

Where:

$\begin{matrix}{{F_{m}^{t}\left( \overset{\rightarrow}{x} \right)} = {\sum\limits_{\{{{j|t} = {D{(y_{j}^{\overset{\rightarrow}{x}})}}}\}}{L_{j}^{u}{I\left( {\overset{\rightarrow}{z}}_{m}^{{\overset{\rightarrow}{y}}_{j}^{\overset{\rightarrow}{x}}} \right)}}}} & (44)\end{matrix}$

The last term in equation 43 is a class-coupling term, defined as:

$\begin{matrix}{{G_{m}^{t}\left( \left\{ H^{v \neq t} \right\} \right)} = {\sum\limits_{x}{{F_{m}^{t}\left( \overset{\rightarrow}{x} \right)}{\sum\limits_{\{{k|{v \neq t}}\}}{L_{k}^{u}{\sum\limits_{n}{H_{n}^{v}{I\left( {\overset{\rightarrow}{z}}_{n}^{{\overset{\rightarrow}{y}}_{k}^{\overset{\rightarrow}{x}}} \right)}}}}}}}} & (45)\end{matrix}$

Where the decimation class index v in equation in 45 is determined bythe class map as follows.

v=D(y _(h) ^({right arrow over (x)}))   (46)

The solution structure in equation 43 has the form:

$\begin{matrix}{b_{m}^{t} = {{\sum\limits_{n}{H_{n}^{t}A_{n,m}^{t}}} + {G_{m}^{t}\left( \left\{ H^{v \neq t} \right\} \right)}}} & (47)\end{matrix}$

Where the b^(t) term is an observation vector for decimation class t:

$\begin{matrix}{b_{n}^{t} = {\sum\limits_{\overset{->}{x}}{{F_{m}^{t}\left( \overset{->}{x} \right)}{I\left( \overset{->}{x} \right)}}}} & (48)\end{matrix}$

And the A^(t) term is a correlation matrix for decimation class t:

$\begin{matrix}{A_{n,m}^{t} = {\sum\limits_{\overset{->}{x}}{{F_{n}^{t}\left( \overset{->}{x} \right)}{F_{m}^{t}\left( \overset{->}{x} \right)}}}} & (49)\end{matrix}$

The indexes {m, n} run over the decimation filter taps m=0, 1, . . .T−1, n=0, 1 . . . T−1, with T denoting the size of the decimation filtertap. The indexes {j, k} run over the prediction filter taps. Thesolution structure is valid for any three-dimensional classifiedprediction and decimation filter. In other words, it is valid for anynumber of classes, and for any prediction class map u=S({right arrowover (x)}) and decimation class map: v=D({right arrow over (y)}).

The solution for the multiple class decimation filters is given byequation 47, with the definition of the G^(t) term given in equation 45.The G^(t) term couples the filters from different decimation classes.G^(t) is designated a class-coupling term because it does not allow foreach class to be solved independently. The class-coupling term increasesthe complexity extracting the decimation filters.

The function F_(m) ^(t)({right arrow over (x)}) is a quantity thatincorporates the effect of the classified prediction filter. Thisfunction is referred to as the L-weighted input signal and is a sum ofthe input signal I(Z_(m) ^(y) ^(j) _({right arrow over (x)}) ) weightedwith the prediction filter coefficient L_(j) ^(u). This sum is over allthe prediction filter tap positions y_(j) ^({right arrow over (x)}) thatbelong to the same decimation class t=D(y_(j)^({right arrow over (x)})). For a given target pixel {right arrow over(x)}, F_(m) ^(t)({right arrow over (x)}) may be zero for some classes.

The mono-class decimation filter described above primarily inconjunction with FIG. 7 is given by the matrix inversion of thecorrelation matrix H=(A)⁻¹b as follows, where the correlation matrix isdefined:

$\begin{matrix}{A_{n,m} = {\sum\limits_{\overset{->}{x}}{{F_{n}\left( \overset{->}{x} \right)}{F_{m}\left( \overset{->}{x} \right)}}}} & (50)\end{matrix}$

and the observation vector is defined:

$\begin{matrix}{{b_{m} = {\sum\limits_{\overset{->}{x}}{{I\left( \overset{->}{x} \right)}{F_{m}\left( \overset{->}{x} \right)}}}}{where}} & (51) \\{{F_{n}\left( \overset{->}{x} \right)} = {\sum\limits_{j}{L_{j}^{u}{I\left( {\overset{->}{z}}_{n}^{{\overset{->}{y}}_{i}^{\overset{.}{x}}} \right)}}}} & (52)\end{matrix}$

is the L-weighted input signal for the mono-class case. Equation 52 isthe least squares solution for extracting an optimized mono-classdecimation filter, given some classified prediction filter. The leastsquares training solution may include computing the correlation matrixand observation vector over a set of samples, where the set of samplesis all the pixels belonging to the class being considered. In themono-class case, all the pixels are used, since there is only one classfor the entire image. The least squares solution yields a stablesolution as long as the set of samples is large enough for each filtercase.

The class-coupling term does not allow for a direct solution for eachclass. In other words, the decimation filters for each class cannot besolved independently. FIG. 12 illustrates the dependence of theprediction of a pixel from multiple classes. Pixel 1205 is the pixel tobe predicted with a prediction filter using a 9 pixel tap. The pixel tapfor pixel 1205 includes pixels 1210 from a first decimation class,pixels 1215 from a second decimation class, and pixels 1220 from a thirddecimation class. FIG. 12 graphically illustrates the class-couplingterm. Border 1225 divides class 0 from class 1. Border 1230 dividesclass 1 from class 2.

The prediction filter for the target pixel 1205 is expressed as:

$\begin{matrix}{{P\left( \overset{->}{x} \right)} = {{\sum\limits_{j = 0}^{8}{L_{j}^{u}{I^{D}\left( {\overset{->}{y}}_{j}^{\overset{\_}{x}} \right)}}} = {\sum\limits_{j = 0}^{8}{L_{j}^{u}{\sum\limits_{n}{H_{n}^{v}{I\left( {\overset{->}{z}}_{n}^{{\overset{\_}{y}}_{j}^{\overset{->}{x}}} \right)}}}}}}} & (53)\end{matrix}$

where H_(n) ^(v) is the classified decimation Filter for class v at tapposition n. This prediction may be separated into the contributions fromthe different classes shown in FIG. 7 b:

$\begin{matrix}{{P\left( \overset{->}{x} \right)} = {{\sum\limits_{{j = 0},3,6}\; {L_{j}^{u}{\sum\limits_{n}{H_{n}^{0}{I\left( {\overset{->}{z}}_{n}^{{\overset{\_}{y}}_{j}^{\overset{.}{x}}} \right)}}}}} + {\sum\limits_{{j = 4},7,8}\; {L_{j}^{u}{\sum\limits_{n}{H_{n}^{1}{I\left( {\overset{->}{z}}_{n}^{{\overset{\_}{y}}_{j}^{\overset{.}{x}}} \right)}}}}} + {\sum\limits_{{j = 1},2,5}\; {L_{j}^{u}{\sum\limits_{n}{H_{n}^{2}{I\left( {\overset{->}{z}}_{n}^{{\overset{\_}{y}}_{j}^{\overset{.}{x}}} \right)}}}}}}} & (54)\end{matrix}$

this shows how the prediction for each target pixel is influenced bymultiple decimation classes. This is the source of the class couplingterm, G.

The L-weighted input signal F_(m) ^(t)({right arrow over (x)}) fordecimation filter tap position 1235 may be defined as:

$\begin{matrix}{{{F_{m}^{t}\left( \overset{->}{x} \right)} = {\sum\limits_{\{{{jt} = {D{(y_{j}^{\overset{\_}{x}})}}}\}}{L_{j}^{u}{I\left( {\overset{->}{z}}_{m}^{{\overset{\_}{y}}_{j}^{\overset{.}{x}}} \right)}}}}{{where}\text{:}}} & (55) \\{{F_{2}^{0}\left( \overset{->}{x} \right)} = {{L_{0}^{u}{I\left( {\overset{->}{z}}_{2}^{{\overset{\_}{y}}_{0}^{\overset{.}{x}}} \right)}} + {L_{3}^{u}{I\left( {\overset{->}{z}}_{2}^{{\overset{->}{y}}_{3}^{\overset{-}{x}}} \right)}} + {L_{6}^{u}{I\left( {\overset{->}{z}}_{2}^{{\overset{\_}{y}}_{6}^{\overset{.}{x}}} \right)}}}} & (56) \\{{F_{2}^{1}\left( \overset{->}{x} \right)} = {{L_{4}^{u}{I\left( {\overset{->}{z}}_{2}^{{\overset{\_}{y}}_{4}^{\overset{.}{x}}} \right)}} + {L_{7}^{u}{I\left( {\overset{->}{z}}_{2}^{{\overset{->}{y}}_{7}^{\overset{.}{x}}} \right)}} + {L_{8}^{u}{I\left( {\overset{->}{z}}_{2}^{{\overset{\_}{y}}_{8}^{\overset{.}{x}}} \right)}}}} & (57) \\{{F_{2}^{2}\left( \overset{->}{x} \right)} = {{L_{1}^{u}{I\left( {\overset{->}{z}}_{2}^{{\overset{->}{y}}_{1}^{\overset{\_}{x}}} \right)}} + {L_{2}^{u}{I\left( {\overset{->}{z}}_{2}^{{\overset{->}{y}}_{2}^{\overset{-}{x}}} \right)}} + {L_{5}^{u}{I\left( {\overset{->}{z}}_{2}^{{\overset{\_}{y}}_{5}^{\overset{.}{x}}} \right)}}}} & (58)\end{matrix}$

such that:

F ₂ ^(t)({right arrow over (x)})=0 t>2   (59)

where equation 54 represents the contribution of class zero, equation 55represents the contribution of class one, and equation 56 represents thecontribution of class two to the L-weighted input signal.

This is unlike the case of multi-class prediction, where the predictionfilters for each class may be solved independently. For multi-classprediction, the solution is the same matrix structure for each class,similar to the mono-class decimation filter case.

FIG. 13 illustrates multiple decimation filter extraction using an exactprocess. In one embodiment, the method 500 may execute the method 1300at block 520. At block 1305, the method concatenates the decimationfilters for each class into a single combined decimation filter. Thiscombined filter may be obtained exactly from a matrix inverse solution.The combined decimation filter may be defined as:

{H _(n) ^(t=0) , H _(n) ^(t=1) , . . . H _(n) ^(t=M−1) }→Ĥ _(tT+n) =Ĥ_(p)   (60)

The index t runs over the number of classes t=0, 1, 2 . . . M−1, where Mis the total number of classes. The index n runs over the decimationfilter taps n=0, 1, 2 . . . T−1 where T is the total number of taps. Theindex p runs over all the elements of the combined filter p=0, 1, 2 . .. MT−1. Since the combined filter embodies all classes, noclass-coupling term is present. At block 1310, the method generates theL-weighted input signal as follows:

$\begin{matrix}{{{\left\{ {F_{m}^{0},F_{m}^{1},{\ldots \mspace{14mu} F_{m}^{t = {M - 1}}}} \right\}->{\hat{F}}_{p = {{tT} + m}}} = {\hat{F}}_{p}}{where}} & (61) \\{{\hat{F}}_{p} = {{\hat{F}}_{p = {{tT} + m}} = {\sum\limits_{\{{{jt} = {D{(y_{j}^{\overset{.}{x}})}}}\}}{L_{j}^{u}{I\left( {\overset{->}{z}}_{m}^{{\overset{\_}{y}}_{j}^{\overset{.}{x}}} \right)}}}}} & (62)\end{matrix}$

At block 1315, the method extends the observation vector and correlationmatrix into one joint vector and matrix for all classes. The solutionequation for the combined filter has the matrix form:

$\begin{matrix}{{\hat{b}}_{q} = {\sum\limits_{n}{{\hat{H}}_{p}{\hat{A}}_{p,q}}}} & (63)\end{matrix}$

where the observation vector and correlation matrix are defined as:

$\begin{matrix}{{\hat{b}}_{q} = {\sum\limits_{\overset{->}{x}}{{I\left( \overset{->}{x} \right)}{{\hat{F}}_{q}\left( \overset{\_}{x} \right)}}}} & (64) \\{{\hat{A}}_{p,q} = {\sum\limits_{\overset{¨}{x}}{{{\hat{F}}_{p}\left( \overset{\_}{x} \right)}{{\hat{F}}_{q}\left( \overset{\_}{x} \right)}}}} & (65)\end{matrix}$

At block 1320, the method extracts the solution for the combineddecimation filter by direct matrix inversion:

Ĥ=(Â)⁻¹ {circumflex over (b)}  (66)

At block 1325, the method de-concatenates the decimation filtercoefficients for each class H_(n) ^(t) are from the combined filter,

Ĥ _(p) =Ĥ _(tT+n) →{H _(n) ^(t=0) ,H _(n) ^(t=1) , . . . H _(n)^(t=M−1)}  (67)

The class-coupling term is incorporated into the combined L-weightedinput signal. The solution structure for the extended decimation filteris simply a matrix inversion.

The exact solution involves inverting the extended correlation matrix,which has a size of MT×MT. For some cases, the number of taps may exceed100 for a multi-field tap, and the number of classes may exceed 128. Theproduces a large matrix with a correspondingly costly inversionoperation. In certain applications, such as HDTV (high definition TV) orUDTV (ultra definition TV), the complexity cost may exceed availablecomputational resources.

FIG. 14 illustrates another method of extracting multiple decimationfilters according to an embodiment of the invention. In one embodiment,the method 500 may execute the method 1400 at block 520. The methodreduces the complexity of the exact method illustrated in FIG. 13 byonly inverting matrixes for each class. This approximate method treatsthe class-coupling term as a modified observation vector. The methodperforms multiple passes to converge on a solution with lower, butsimilar, performance to the exact method illustrated in FIG. 13.

The solution for the multiple decimation filters may be defined as:

$\begin{matrix}{b_{m}^{t} = {{\sum\limits_{n}{H_{n}^{t}A_{n,m}^{t}}} + {G_{m}^{t}\left( \left\{ H^{v \neq t} \right\} \right)}}} & (68)\end{matrix}$

At block 1405, the method generates a correlation matrix and observationvector for each class. At block 14 10, the method generates some initialvalues of the decimation filters (described below in greater detail inconjunction with FIG. 15), and incorporates the class coupling term as amodification of the observation vector:

{tilde over (b)} _(m) ^(t) =b _(m) ^(t) −G _(m) ^(t)({{tilde over (H)}^(v≠t)})   (69)

The solution for the multiple declination filters takes the form:

$\begin{matrix}{{\overset{\sim}{b}}_{m}^{t} = {\sum\limits_{n}{H_{n}^{t}A_{n,m}^{t}}}} & (70)\end{matrix}$

The method obtains the solution for the decimation filter for each classt as:

H ^(t)=(A ^(t))⁻¹ {tilde over (b)} ^(r)   (71)

At block 1415, the method uses the decimation filter for each class toupdate the class-coupling term. At block 1420, the method generates amodified observation vector. At block 1425, the method uses the modifiedobservation vector to obtain a new solution for the classifieddecimation filters. At block 1430, the method determines the predictionerror given the new decimation filters. At block 1435, the methodrepeats this process for some number of iterations. Empirical dataindicates that the prediction improvement saturates after 2-3iterations. The prediction error at each stage may be computed toevaluate the effectiveness of the decimation filters at each pass. Thecomplexity for this approximate process is order ˜2M(T)^(λ) where λ is anumber between ˜2.4 and 3 depending on the matrix inversion method. Thiscompares favorably with extraction method illustrated in FIG. 13, whichhas an order of complexity of ˜(MT)^(λ), especially for large number ofclasses M.

FIG. 15 illustrates a method of estimating initial decimation filtersaccording to an embodiment of the invention. The prediction for a targetpixel may involve multiple decimation classes, because the decimationclass map is defined on the decimated data as

v=D(y _(j) ^({right arrow over (x)}))   (72)

where {right arrow over (x)} is the target pixel and {j} is theprediction filter tap index. As the tap index j runs over the predictionfilter tap, each decimated data point y_(j) ^({right arrow over (x)})may have a different class assigned to it, from the class map D. Themultiple classes associated with the target pixel generates theclass-coupling term.

The method introduces an initial estimation of the decimation filterswith a map that assigns each target pixel to one decimation class. Themethod may use different types of assignments. At block 1505, the methodobtains a target pixel-to-decimation class map as follows:

v=R({right arrow over (x)})   (73)

For example:

R({right arrow over (x)})=D(y _(j) _(select) ^({right arrow over (x)}))  (74)

where

j _(select)=sup_(j) {|L _(j) ^(u)|}  (75)

The target pixel-to-decimation map means that for the target pixel{right arrow over (x)}, the method uses the class associated with thedecimated data y_(j) ^({right arrow over (x)}) that has the largestprediction filter coefficient. The largest prediction filter coefficientis found over all the prediction filter tap positions indexed by j. Inother embodiments, other definitions for associating a decimation classwith a target pixel may be used. At block 1510, the method generates amodified input signal from image and prediction filters. The methodgenerates a modified input signal for each class.

The method uses the target pixel-to-decimation class map to simplify theextraction of the decimation filters. The class coupling term is zero,and the solution has the form:

$\begin{matrix}{b_{m}^{t} = {\sum\limits_{n}{H_{n}^{t}A_{n,m}^{t}}}} & (76)\end{matrix}$

At block 1515, the method modifies the correlation matrix andobservation vector to contain the constraint from the map R({right arrowover (x)}). The modification is in the summation over the target pixels.The summation is now only over a partition of the full input space. Inother words, only target pixels corresponding to a specific decimationclass are summed.Mathematically, the method modifies the observation vector andcorrelation matrix as follows:

$\begin{matrix}{b_{n}^{t} = {\sum\limits_{\{{{\overset{->}{x}t} = {R(\overset{->}{x})}}\}}{{I\left( \overset{->}{x} \right)}{F_{m}^{t}\left( \overset{->}{x} \right)}}}} & (77) \\{A_{n,m}^{t} = {\sum\limits_{\{{{\overset{->}{x}t} = {R(\overset{->}{x})}}\}}{{F_{n}^{t}\left( \overset{->}{x} \right)}{F_{m}^{t}\left( \overset{->}{x} \right)}}}} & (78)\end{matrix}$

where the modified L-weighted input signal is defined as:

$\begin{matrix}{{F_{m}^{t}\left( \overset{->}{x} \right)} = {\sum\limits_{\{{j_{select}(\overset{->}{x})}\}}{L_{j}^{u}{I\left( {\overset{->}{z}}_{m}^{{\overset{->}{y}}_{j}^{\overset{->}{x}}} \right)}}}} & (79)\end{matrix}$

At block 1520, the method obtains the solution for the multipledecimation filters as follows:

H ^(t)=(A ^(t))⁻¹ b ^(t)   (80)

and extracts an initial decimation filter for each class from themodified correlation matrix and the modified correlation vector.

In one embodiment, as shown in FIG. 11A, a server computer 1101 iscoupled to, and provides data through, the Internet 1105. A clientcomputer 1103 is coupled to the Internet 1105 through an ISP (InternetService Provider) 1105 and executes a conventional Internet browsingapplication to exchange data with the server 1101. Optionally, theserver 1101 can be part of an ISP which provides access to the Internetfor client systems. The term “Internet” as used herein refers to anetwork of networks which uses certain protocols, such as the TCP/IPprotocol, and possibly other protocols such as the hypertext transferprotocol (HTTP) for hypertext markup language (HTML) documents that makeup the World Wide Web (web). The physical connections of the Internetand the protocols and communication procedures of the Internet are wellknown to those of skill in the art. Access to the Internet allows usersof client computer systems to exchange information, receive and sende-mails, view documents, such as documents which have been prepared inthe HTML format, and receive content. It is readily apparent that thepresent invention is not limited to Internet access and Internetweb-based sites; directly coupled and private networks are alsocontemplated.

One embodiment of a computer system suitable for use as server 1101 isillustrated in FIG. 11B. The computer system 1110, includes a processor1120, memory 1125 and input/output capability 1130 coupled to a systembus 1135. The memory 1125 is configured to store instructions which,when executed by the processor 1120, perform the methods describedherein. The memory 1125 may also store data for adaptive decimationfiltering. Input/output 1130 provides for the delivery and display ofthe data for adaptive decimation filtering or portions orrepresentations thereof, and also the input of data of various types forstorage, processing or display. Input/output 1130 also encompassesvarious types of machine-readable storage media, including any type ofstorage device that is accessible by the processor 1120. One of skill inthe art will immediately recognize that the server 1101 is controlled byoperating system software executing in memory 1125. Input/output 1130and related media store the machine-executable instructions for theoperating system and methods of the present invention as well as thedata for an adaptive decimation filter.

The description of FIGS. 11A-B is intended to provide an overview ofcomputer hardware and other operating components suitable forimplementing the invention, but is not intended to limit the applicableenvironments. It will be appreciated that the computer system 1140 isone example of many possible computer systems which have differentarchitectures. A typical computer system will usually include at least aprocessor, memory, and a bus coupling the memory to the processor. Oneof skill in the art will immediately appreciate that the invention canbe practiced with other computer system configurations, includingmultiprocessor systems, minicomputers, mainframe computers, and thelike. The invention can also be practiced in distributed computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network.

An adaptive decimation filter has been described. Although specificembodiments have been illustrated and described herein it will beappreciated by those of ordinary skill in the art that any arrangementwhich is calculated to achieve the same purpose may be substituted forthe specific embodiments shown. This application is intended to coverany adaptations or variations of the present invention.

1. A computerized method comprising: concatenating a plurality ofinitial decimation filters into an initial combined decimation filter,wherein the plurality of initial decimation filters are assigned to aplurality of classes, the plurality of classes associating the pluralityof initial decimation filters with a plurality of regions of an image tobe predicted; generating an input signal from the initial combineddecimation filter, wherein the input signal is weighted with a pluralityof prediction filters; generating a correlation matrix for the initialcombined decimation filter from the input signal; generating anobservation vector for the initial combined decimation filter from theinput signal; extracting an optimized combined decimation filter fromthe correlation matrix and the observation vector, the optimizedcombined decimation filter comprising a plurality of optimizeddecimation filters; and de-concatenating the optimized combineddecimation filter into the plurality of optimized decimation filters. 2.A computer-readable storage medium storing instructions which whenexecuted by a data processing system cause the data processing system toperform operations that process data, the operations comprising:concatenating a plurality of initial decimation filters into an initialcombined decimation filter, wherein the plurality of initial decimationfilters are assigned to a plurality of classes, the plurality of classesassociating the plurality of initial decimation filters with a pluralityof regions of an image to be predicted; generating an input signal fromthe initial combined decimation filter, wherein the input signal isweighted with a plurality of prediction filters; generating acorrelation matrix for the initial combined decimation filter from theinput signal; generating an observation vector for the initial combineddecimation filter from the input signal; extracting an optimizedcombined decimation filter from the correlation matrix and theobservation vector, the optimized combined decimation filter comprisinga plurality of optimized decimation filters; and de-concatenating theoptimized combined decimation filter into the plurality of optimizeddecimation filters.
 3. A system comprising: a processor coupled to abus; a memory coupled to the processor through the bus, and instructionsexecuted by the processor from the memory to cause the processor toconcatenate a plurality of initial decimation filters into an initialcombined decimation filter, wherein the plurality of initial decimationfilters are assigned to a plurality of classes, the plurality of classesassociating the plurality of initial decimation filters with a pluralityof regions of an image to be predicted; generate an input signal fromthe initial combined decimation filter, wherein the input signal isweighted with a plurality of prediction filters; generate a correlationmatrix for the initial combined decimation filter from the input signal;generate an observation vector for the initial combined decimationfilter from the input signal; extract an optimized combined decimationfilter from the correlation matrix and the observation vector, theoptimized combined decimation filter comprising a plurality of optimizeddecimation filters; and de-concatenate the optimized combined decimationfilter into the plurality of optimized decimation filters.
 4. A dataprocessing system comprising: means for concatenating a plurality ofinitial decimation filters into an initial combined decimation filter,wherein the plurality of initial decimation filters arc assigned to aplurality of classes, the plurality of classes associating the pluralityof initial decimation filters with a plurality of regions of an image tobe predicted; means for generating an input signal from the initialcombined decimation filters wherein the input signal is weighted with aplurality of prediction filters; means for generating a correlationmatrix for the initial combined decimation filter from the input signal;means for generating an observation vector for the initial combineddecimation filter from the input signal; means for extracting anoptimized combined decimation filter from the correlation matrix and theobservation vector the optimized combined decimation filter comprising aplurality of optimized decimation filters; and means forde-concatenating the optimized combined decimation filter into theplurality of optimized decimation filters.
 5. A computerized methodcomprising: generating a correlation matrix and observation vector foran image; initializing a plurality of decimation filters correspondingto a plurality of classes classifying the image; for each class in theplurality of classes, generating a class-coupling term using theplurality of decimation filters; generating a modified observationvector using the class-coupling term and the plurality of decimationfilters; and for each class in the plurality of classes, replacing thedecimation filter corresponding to the class with an optimizeddecimation filter extracted from the correlation matrix and the modifiedobservation vector.
 6. The computerized method of claim 5 whereininitializing the plurality of decimation filters comprises: generating atarget pixel-to-class map; for each class in the plurality of classes,generating a modified input signal from the image and prediction filtersassociated with the image; generating a modified correlation matrix andobservation vector for each class using the modified input signal andthe image; and extracting a decimation filter for each class in theplurality of classes based on the modified correlation matrix and themodified observation vector.
 7. A computer-readable storage mediumstoring instructions which when executed by a data processing systemcause the data processing system to perform operations that processdata, the operations comprising: generating a correlation matrix andobservation vector for an image; initializing a plurality of decimationfilters corresponding to a plurality of classes classifying the image;for each class in the plurality of classes, generating a class-couplingterm using the plurality of decimation filters; generating a modifiedobservation vector using the class-coupling term and the plurality ofdecimation filters; and for each class in the plurality of classes,replacing the decimation filter corresponding to the class with anoptimized decimation filter extracted from the correlation matrix andthe modified observation vector.
 8. The computer-readable storage mediumof claim 7 wherein initializing the plurality of decimation filterscomprises: generating a target pixel-to-class map; for each class in theplurality of classes, generating a modified input signal from the imageand prediction filters associated with the image; generating a modifiedcorrelation matrix and observation vector for each class using themodified input signal and the image; and extracting a decimation filterfor each class in the plurality of classes based on the modifiedcorrelation matrix and the modified observation vector.
 9. A systemcomprising: a processor coupled to a bus; a memory coupled to theprocessor through the bus; and instructions executed by the processorfrom the memory to cause the processor to generate a correlation matrixand observation vector for an image; initialize a plurality ofdecimation filters corresponding to a plurality of classes classifyingthe image; for each class in the plurality of classes, generate aclass-coupling term using the plurality of decimation filters; generatea modified observation vector using the class-coupling term and theplurality of decimation filters; and for each class in the plurality ofclasses replace the decimation filter corresponding to the class with anoptimized decimation filter extracted from the correlation matrix andthe modified observation vector.
 10. The system of claim 5 whereininitializing the plurality of decimation filters comprises: generate atarget pixel-to-class map; for each class in the plurality of classes,generate a modified input signal from the image and prediction filtersassociated with the image; generate a modified correlation matrix andobservation vector for each class using the modified input signal andthe image; and extract a decimation filter for each class in theplurality of classes based on the modified correlation matrix and themodified observation vector.
 11. A data processing system comprising:means for generating a correlation matrix and observation vector for animage; means for initializing a plurality of decimation filterscorresponding to a plurality of classes classifying the image; for eachclass in the plurality of classes, means for generating a class-couplingterm using the plurality of decimation filters; means for generating amodified observation vector using the class-coupling term and theplurality of decimation filters; and for each class in the plurality ofclasses, means for replacing the decimation filter corresponding to theclass with an optimized decimation filter extracted from the correlationmatrix and the modified observation vector.
 12. The data processingsystem of claim 11 wherein initializing the plurality of decimationfilters comprises: means for generating a target pixel-to-class map; foreach class in the plurality of classes, means for generating a modifiedinput signal from the image and prediction filters associated with theimage; means for generating a modified correlation matrix andobservation vector for each class using the modified input signal andthe image; and means for extracting a decimation filter for each classin the plurality of classes based on the modified correlation matrix andthe modified observation vector.